Repository navigation
Shard offline Stim sample files across ranks and reduce pymatching_only counts - #100
Conversation
There was a problem hiding this comment.
Thanks for the contribution and the detailed reproduction.
The diagnosis is correct: under torch.distributed every rank slices the same leading rows of the .dets file, and pymatching_only returns before the all-reduce. Sharding in the datapipe plus the reduce in pymatching_only addresses both, and single-process behaviour is unchanged (rank=0, world_size=1 reduces to the previous slice). Sharding is preferred over raising for world_size > 1.
Requested changes are inline. One item outside the diff: the README section "Decode the files" should state the multi-GPU behaviour (contiguous per-rank blocks, what happens to remainder shots, and what the PREDECODER_DECODE_OUTPUT_DIR arrays contain).
|
Thanks for the review. All points are in d0f56e8:
One note: the old gloo test compared against the matcher in |
|
/ok to test d0f56e8 |
ivanbasov
left a comment
There was a problem hiding this comment.
Thanks for the update. The follow-up commit addresses all four inline comments and the README. Two remaining items:
_stim_file_shard: whenrequested_per_rank == file_shots // world_sizeand the file size is not divisible byworld_size, the whole file is decoded, which is more thantest.num_samples(11-shot file,num_samples=10, 2 ranks: 11 shots, versus 10 on one rank). Please cap the total atmin(test.num_samples, file_shots)so the multi-rank shot count matches the single-rank one in this case as well, and extend the shard tests accordingly._gather_decode_rowsis exercised on the gloo backend only. Has thePREDECODER_DECODE_OUTPUT_DIRpath been run on a multi-GPU host with NCCL? If so, please add the command and result to the PR description.
…ly counts With PREDECODER_STIM_SAMPLES_DIR set and more than one rank, every rank built its dataset from the leading rows of the .dets file, so all ranks decoded the same block and the all-reduce counted it world_size times. pymatching_only also returned before the all-reduce, so rank 0 reported only its own shard. Give each rank a contiguous, disjoint block of the file (rank and world_size passed through DatapipeFactory), and all-reduce the pymatching_only error and shot counts like the pre-decoder path does. Signed-off-by: Dezhen Lu <dezhen.lu@student.uni-tuebingen.de>
With an output directory set, rank 0 wrote only its own block of shots, so the saved arrays did not cover the shots behind the reported counts. Gather the observables, PyMatching predictions, pre-decoder predictions and residual detectors on rank 0 in rank order before saving them. When num_samples does not cap the shard, split the whole Stim file and give the first file_shots % world_size ranks one extra shot, so a file yields the same shot count on N ranks as on one. When it does cap the shard, rank 0 prints how many of the file's shots are decoded. A file with fewer shots than ranks is rejected on every rank. Move the test config helper to module level, add negative tests for the two ValueError paths and compare a two-rank gloo run against one process, and document the multi-GPU behaviour in the README. Signed-off-by: Dezhen Lu <dezhen.lu@student.uni-tuebingen.de>
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
d0f56e8 to
928fb47
Compare
|
Rebased onto main and capped the selected prefix before splitting it across ranks. The offline suite passed, including the 11-shot file capped at 10 and the two-process Gloo output comparison. I have not run the output path with multi-GPU NCCL. |
|
/ok to test 928fb47 |
|
Merged. Thank you for the contribution and for the quick turnaround on the review comments. |
Offline decoding from
.detsfiles (PREDECODER_STIM_SAMPLES_DIR) reports wrong LER statistics when inference runs with more than one rank.code/scripts/local_run.shuses every visible GPU and launchestorch.distributed.runwhen there is more than one, and the offline commands in the README do not setGPUS, so this happens on any multi-GPU host unless the user passesGPUS=1(onlyoffline_smoketest.shdoes).There are two problems:
num_samples // world_sizeshots, butQCDataPipePreDecoder_Memory_from_stim_filealways keeps the leading rows of the file. Every rank decodes the same first block. Withising_decoding_pymatching, the all-reduce then counts those shotsworld_sizetimes and the rest of the file is never decoded. The reported shot count looks right, so nothing hints at the problem, and the standard error is too small by a factor of sqrt(world_size). The syndrome-density reduction path builds the dataset the same way and all-reduces identical shards too; decoder ablation also builds it the same way and reports rank 0's block. Both go through the factory, so they get disjoint shards with this change as well.pymatching_onlyreturns before the all-reduce, so rank 0 reports only its own shard (num shots=num_samples // world_size).Repro: d=5, n_rounds=5, O1, default 25-parameter noise model, 262,144 shots per basis written by
workflow.task=generate_stim_data, thenPREDECODER_DECODE_MODE=pymatching_only WORKFLOW=inference, CPU with the gloo backend (same code path as NCCL):Decoding the same files directly with
stim.read_shot_data_fileand PyMatching on the circuit's DEM gives 1403 / 1377 errors over the whole file, 687 / 682 on the first half and 716 / 695 on the second half. The 2-rank numbers on main are 687/131072 and 682/131072: rank 0's half only, and rank 1 decoded the same shots. With 4 ranks main reports 0.005203 / 0.005585, the first quarter of the file (341 / 366 errors times 4); with this change 4 ranks also give 0.005352 / 0.005253.Sharding
The file-backed datapipe applies the global shot limit before splitting the selected rows into contiguous blocks. The first ranks receive one extra shot if needed. Both decode modes reduce their counts across ranks. When decode output is enabled, rank 0 gathers the arrays in file order.
Validation
PYTHONPATH=code python -m pytest -q code/tests/test_offline_stim_decoding.pypassed all 25 tests, including the 11-shot file capped at 10 and the two-process Gloo comparison of saved arrays against a single process. The failure-analysis suite passed all 77 tests. Multi-GPU NCCL remains untested.